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COMMUNICATION SYSTEM, COMMUNICATION APPARATUS, 
AND COMMUNICATION METHOD 

BACKGROUND OF THE INVENTION 
5 Field of the Invention 

The present invention relates to a system, an 
apparatus, and a method for remotely operating a 
desired electronic device by means of a graphical user 

Q 

interface. 
10 Related Background Art 
Cj In recent years there have been developed high- 

™4 speed and advanced digital interfaces for transmitting 

L data expected to have real time properties such as a 

f=f dynamic image (moving image) or a sound or voice and 

5^ 15 data expected to have reliability on communications 

=^ such as a static image and a control command. A 

digital interface conforming to the IEEE1394-1995 
standards is one of these digital interfaces. 

Furthermore recently there has been suggested that 
2 0 electronic devices in a house having this type of 
digital interfaces (for example, AV (audio-visual) 
equipment, household electric appliances, and a 
personal computer, etc.) are connected with each other 
so as to construct a home network. 
2 5 When the home network prevails among us, a device 

functioning as a center of the home network is, for 
example, a television set. If so, the television set 
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provides a user with graphical user interfaces for 
remotely operating other devices and the user operates 
a graphical user interfaces displayed on the television 
set to operate a device corresponding to the graphical 
5 user interface. 

There, however, are a plurality of types of 
devices connectable to the home network and they have 
...J operating environments different from each other. 

Furthermore, even if the devices have the same 
10 function, their operating environments depend upon 
N! their manufacturers or model types. 

03 Therefore, there is such a problem that the 

p television set is required to cope with graphical user 

bi 

ui interfaces depending upon manufacturers or model types 

^ 15 flexibly. 

" In addition, there is also a problem that the 

different operation methods for each graphical user 
interface will cause an operation to be complicated. 
To solve this problem, there can be a method of 

20 operating each graphical user interface by using 
direction keys and selection keys . 

This method, however, has a problem of requiring 
the direction keys to be operated by a plurality number 
of times until a desired function is selected, by which 

25 it takes much time for executing the desired function. 

Particularly if a plurality of operations are required 
to be executed continuously, the operations are 
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complicated, by which quick operations cannot be 
performed disadvantageously . 

SUMMARY OF THE INVENTION 
5 An object of the present invention is to solve the 

above-described problems. 

According to one aspect, the present invention 
which achieves this object related to a communication 
system, comprising: a controlled device for managing an 

10 operation panel having a plurality of display elements 
corresponding to predetermined operations, a controller 
for accepting an operation for the operation panel 
supplied by the controlled device, and an operation 
apparatus having a first operation unit for operating 

15 the operation panel and a second operation unit for 
specifying a specific operation, wherein said 
controller transmits a first control signal indicating 
that any of said display element has been operated on 
the basis of an operation for the first operation unit 

20 to the controlled device and transmits a second control 
signal for specifying any of the specific operations on 
the basis of an operation for the second operation unit 
to the controlled device. 

According to another aspect, the present invention 

25 which achieves this object related to a communication 
apparatus, comprising: operation panel managing means 
for managing an operation panel having a plurality of 



display elements corresponding to predetermined 
operations and for supplying the operation panel to a 
controller for accepting an operation for the operation 
panel, receiving means for receiving a first control 
signal indicating that the display element has been 
operated and a second control signal for specifying any 
of specific operations from the controller, and control 
means for controlling an operation corresponding to the 
display element to be executed on the basis of the 
first control signal and controlling the specific 
operation to be executed on the basis of the second 
control signal. 

According to still another aspect, the present 
invention which achieves this object related to a 
communication method, comprising the steps of: 
supplying an operation panel having a plurality of 
display elements corresponding to predetermined 
operations to a controller for accepting an operation 
for the operation panel, and receiving a first control 
signal indicating that any of the display elements has 
been operated or a second control signal for specifying 
any of specific operations, wherein a control is made 
so that an operation corresponding to the display 
element is executed if the first control signal is 
received and the specific operation is executed if the 
second control signal is received. 

According to another aspect, the present invention 
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which achieves this object related to a communication 
apparatus, comprising: receiving means for receiving a 
signal from an operation apparatus having a first 
operation unit for operating an operation panel having 
5 a plurality of display elements corresponding to 

predetermined operations supplied from a controlled 
device and a second operation unit for specifying 
specific operations, and transmitting means for 
transmitting a first control signal indicating that any 

10 of the display elements has been operated on the basis 
of an operation for the first operation unit to the 
controlled device and for transmitting a second control 
signal for specifying any of the specific operations on 
the basis of an operation for the second operation unit 

15 to the controlled device. 

According to still another aspect, the present 
invention which achieves this object related to a 
communication apparatus, comprising: first operation 
means for operating an operation panel having a 

20 plurality of display elements corresponding to 

predetermined operations supplied from a controlled 
device to a controller, second operation means for 
specifying any of specific operations, and transmitting 
means for transmitting a signal indicating that the 

25 first operation means has been operated and a signal 
indicating that the second operating means has been 
operated to the controller. 



- 6 - 



Still other objects of the present invention, and 
the advantages thereof, will become fully apparent from 
the following detailed description of the embodiments. 

5 BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram of assistance in 
explaining an example of a remote control system 
according to an embodiment of the present invention; 
Fig. 2 is a diagram showing an example of an 
10 operation panel according to the embodiment of the 
J] present invention; 

yi Fig. 3 is a diagram showing an example of a remote 

^ control according to the embodiment of the present 

yi invention; 

'^t 15 Fig. 4 is a diagram of assistance in explaining an 

example of a data structure of the operation panel 
according to the embodiment of the present invention; 

Fig. 5 is a diagram of assistance in explaining an 
example of a data structure of a panel element 
20 according to the embodiment of the present invention; 

Fig. 6 is a diagram of assistance in explaining an 
example of a protocol for a remote control system 
according to an embodiment of the present invention; 
Figs. 7A and 7B are diagrams of assistance in 
25 explaining an example of a frame structure of a control 
command and a response; 

Figs. 8A and 8B are diagrams of assistance in 
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explaining a command frame of a GUI_UPDATE command 
according to the embodiment of the present invention; 

Figs. 9A, 9B and 9C are diagrams of assistance in 
explaining a command frame of a PUSH_GUI_DATA command 
5 according to the embodiment of the present invention; 

Fig. 10 is a diagram of assistance in explaining a 
command frame of a USER_ACTION command according to the 
embodiment of the present invention; 
D Fig. 11 is a diagram of assistance in explaining a 

m 10 command frame of a DIRECT_ACTION command according to 

■^n the embodiment of the present invention; 

Fig, 12 is a diagram showing an example of 
action_code; 

r] Fig. 13 is a diagram of assistance in explaining a 

15 processing procedure of a controlled device according 
to an embodiment of the present invention; and 

Fig. 14 is a diagram of assistance in explaining a 
processing procedure of a controller according to the 
embodiment of the present invention. 

20 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The preferred embodiments of the present invention 
will now be described in detail hereinafter with 
reference to the accompanying drawings. 
25 Referring to Fig. 1, there is shown a block 

diagram of assistance in explaining an example of a 
remote control system according to this embodiment. 
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In this system, reference numeral 1 designates a 
digital communication network. This network 1 is 
realized by a high-speed serial bus conforming to the 
IEEE1394-1995 standards and its extended standards 
5 (hereinafter, referred to as IEEE1394 standards). 

Reference numeral 100 designates a camera- 
integrated digital video recorder (hereinafter, a DVC) 
which is an example of a controlled device in this 
O system. The DVC 100 has an operation panel for 

P3 10 providing an operating environment characteristic of 

pa 

® the DVC 100. 

yi Reference numeral 110 designates a tuner that is 

an example of a controlled device in this system. The 
tuner 110 has an operation panel for providing an 

% 15 operating environment characteristic of the tuner 110 

™f in the same manner as the DVC 100. 

The operation panel (also referred to as an 
operation screen) comprises an image or a text and it 
is a graphical user interface providing a user with a 
20 graphical operating environment easy to understand 
visually. 

Reference numeral 120 designates a digital 
television set (hereinafter, a digital television) 
which is an example of a controller in this system. 
25 The digital television 120 displays an operation panel 
of each controlled device to provide a graphical 
operating environment . 



- 9 - 



Reference numeral 130 designates a remote 
controller (hereinafter, a remote control) for 
operating the operation panel displayed on the digital 
television 120. The remote control 130 converts an 
5 operation input of a user to a predetermined control 
signal and wirelessly transmits the control signal to 
the digital television 120. The user can operate the 
operation panel displayed on the digital television 120 
CJ by using the remote control 130 to remotely operate a 

03 10 controlled device corresponding to the operation panel. 

iJJ First, a configuration of the DVC 100 will be 

ISl described below. There are shown a digital interface 

Oj 

„ (hereinafter, a 1394 interface) conforming to the 

r!" IEEE1394 standards, a control unit 102, a Panel 

15 Management Department 103, and a DVC unit 104. 

The Panel Management Department 103 stores GUI 
(graphical user interface) data forming an operation 
panel of the DVC 100 in accordance with a data 
structure described later. If an operation intended 
20 for the operation panel is detected, the Panel 

Management Department 103 reads out (or generates) GUI 
data corresponding to the operation and supplies it to 
the 1394 interface 101. 

The DVC unit 104 records stream data (including 
25 digital video data, digital audio data, and text data) 
conforming to a predetermined data format on a storage 
medium 105 or reproduces desired stream data required 
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by a user from the storage medium 105. The storage 
medium 105 comprises a magnetic tape, a magnetic disk, 
an optical disk, a semiconductor memory, and a hard 
disk or their combinations and is removable. As a 
5 predetermined data foormat, the DV mode, the MPEG2 mode, 
or other modes are used. The stream data recorded on 
the storage medium 105 by the DVC unit 104 is generated 
on the basis of an isochronous packet received by the 
S 139 4 interface 101 and the stream data reproduced from 

yj 10 the storage medium 105 by the DVC unit 104 is supplied 

to the 1394 interface 101. 
01 The 1394 interface 101 is connected to a 1394 

interface of another device via the network 1. The 

St 

r's 1394 interface 101 generates the isochronous packet on 

.nj 

15 the basis of the stream data supplied by the DVC unit 
'r"^ 104 and isochronously transfers this isochronous 

packet. In addition the 1394 interface 101 receives an 
asynchronous packet including a control command for 
controlling operations of the Panel Management 

20 Department 103, the DVC unit 104 and the like and 

supplies the control command to the control unit 102. 
Furthermore, the 1394 interface 101 generates an 
asynchronous packet on the basis of the GUI data 
supplied by the Panel Management Department 103 and 

25 asynchronously transfers this asynchronous packet. 

The control unit 102 has communications via the 
network 1 to control the operations of the DVC unit 104 



and the Panel Management Department 103. The control 
unit 102 reads out a program for realizing a processing 
procedure of this embodiment from the internal storage 
medium and integrally controls respective units of the 
DVC 100 on the basis of the program. 

Next, a configuration of the tuner 110 will be 
described below. There are shown a 1394 interface 111, 
a control unit 112, a Panel Management Department 113, 
and a tuner unit 114. 

The Panel Management Department 113 stores GUI 
data forming an operation panel of the tuner 110 and 
stores it in accordance with a data structure described 
later. If an operation intended for the operation 
panel is detected, the Panel Management Department 113 
reads out (or generates) GUI data corresponding to the 
operation and supplies it to the 1394 interface 111. 

The tuner unit 114 selectively receives a desired 
digital broadcasting wave from a communication line 
network 115, selects desired stream data (including 
digital video data, digital audio data, and text data) 
from the received digital broadcasting wave, and 
supplies the selected stream data to the 1394 interface 
111. The communication line network 115 comprises a 
ground-based broadcasting network, a satellite 
broadcasting network, a cable television broadcasting 
network, a public line network, and a mobile 
communication network or their combinations. 



- 12 - 



The 1394 interface 111 is connected to a 1394 
interface of another device via the network 1 . The 
1394 interface 111 generates an isochronous packet on 
the basis of the stream data supplied by the tuner unit 
5 114 and isochronously transfers the isochronous packet. 
In addition the 1394 interface 111 receives an 
asynchronous packet including a control command for 
controlling operations of the Panel Management 
CJ Department 113, the tuner unit 114, and the like and 

Oi 10 supplies this control command to the control unit 112. 

Ji Furthermore, the 1394 interface 111 generates an 

HI asynchronous packet on the basis of the GUI data 

2~ supplied by the Panel Management Department 113 and 

f;^l asynchronously transfers this asynchronous packet, 

^j; 15 The control unit 112 has communications via the 

r'" network 1 to control the operations of the tuner unit 

114 and the Panel Management Department 113. The 
control unit 112 reads out a program for realizing a 
processing procedure of this embodiment from the 
20 internal storage medium and integrally controls 

respective units of the tuner unit 114 on the basis of 
the program. 

Next, a configuration of the digital television 
120 will be described below. There are shown a 1394 
25 interface 121, a control unit 122, a stream data- 
processing unit 123, a Panel Management Department 124, 
a drawing control unit 12 5, a remote control control 
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unit 126, an image composition unit 127, a display unit 
128, a voice output unit comprising a speaker 129. 

The stream data-processing unit 123 generates 
stream data from the isochronous packet received by the 
5 1394 interface 121 and then separates this stream data 
into video stream data and audio stream data. The 
stream data-processing unit 123 further decodes each 
stream data to supply the video data to the image 
composition unit 12 7 and to supply the audio data to 
10 the voice output unit 129. 

The voice output unit 129 digital-analog-converts 
the audio data supplied by the stream data-processing 
unit 123 and amplifies it to be output. 

The Panel Management Department 12 4 stores the 
15 operation panels which have been transferred from 

respective controlled devices and manages them for each 
controlled device. 

The drawing control unit 125 controls drawings of 
the respective operation panels- Specifically it 
20 controls a display position, a display size, a display 
time, rewriting of displayed contents, a focal 
navigation and the like of each operation panel. 

The image composition unit 127 synthesizes the 
video data supplied by the stream data-processing unit 
25 123 with the operation panel supplied by the drawing 
control unit 125 and digital-analog-converts the 
composed video data so as to be supplied to the display 



- 14 - 



unit 128. 

The display unit 128 comprises a predetermined 
display and displays a video signal supplied by the 
image composition unit 12 7 on this display. The 
5 predetermined display is a CRT (cathode-ray tube), a 
liquid crystal panel, a plasma display panel, a 
pro jection- type display panel, or the like. 

The remote control control unit 126 communicates 
with the remote control 130 via a transmission medium 

10 such as an infrared ray or a radio wave. The remote 
control unit 126 receives a control signal from the 
remote control 130 and supplies this control signal to 
the control unit 122. 

The 1394 interface 121 is connected to a 1394 

15 interface of another device via the network 1. The 
13 9 4 interface 121 receives an asynchronous packet 
including GUI data and supplies this GUI data to the 
Panel Management Department 124 and the drawing control 
unit 125. The 1394 interface 121 asynchronously 

20 transfers a control command obtained by operating the 
remote control 130 to a controlled device. 
Furthermore, the 1394 interface 121 receives an 
asynchronous packet including a control command for 
controlling operations of the stream data-processing 

25 unit 123, the Panel Management Department 124, the 

drawing control unit 12 5 and the like and supplies this 
control command to the control unit 122, 
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The control unit 122 reads out a program for 
realizing a processing procedure of this embodiment 
from the internal storage medium and integrally 
controls respective units of the digital television 120 
5 on the basis of the program. 

Basic functions of the 1394 interfaces 101, 102, 
and 103 will be described below. 

These 1394 interfaces are capable of coping with a 
CJ daisy-chain connection, a node branch connection, and 

05 10 their combinations, thus enabling connections having a 

^ high degree of freedom. Respective transfer rates are 

Hi 100 Mbps, 200 Mbps, and 400 Mbps and data is 

transferred in a serial transfer mode. An interface 

H 

having a high-order transfer rate is free to be 
15 connected to interfaces having their maximum transfer 
rates different from each other so as to support low- 
order data transfer rates. 

In addition, the 1394 interfaces support two 
transfer modes; an asynchronous transfer mode and an 
20 isochronous transfer mode. The asynchronous transfer 

mode is appropriate for transferring a control command, 
file data or other data. The isochronous transfer mode 
is used for ensuring a certain amount of a data 
transfer per predetermined transfer cycle (a single 
25 transfer cycle is approx. 125 ps). The isochronous 
transfer mode is appropriate for transferring data 
expected to have a temporal continuity (what is called 
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stream data) such as video data and audio data. In a 
single transfer cycle period, the isochronous transfer 
mode has a higher priority than the asynchronous 
transfer mode- 
5 (Description of operation panel) 

Referring to Fig. 2, there is shown a diagram of a 
sample operation panel according to this embodiment . 
In Fig. 2, an example is given for a description of 
displaying an operation panel of the DVC 100 which is 

10 one of the controlled devices on the display unit 128 
of the digital television 120, for example. 

In Fig. 2, there are shown a display screen 200 
for the display unit 128, stream data 201 isochronously 
transferred by the DVC 100, and an operation panel as a 

15 panel element 202 for the DVC 100. 

Reference numerals 203 to 207 designate button 
elements- The button element 203, the button element 
204, the button element 205, the button element 206, 
and the button element 20 7 correspond to a rewind 

20 function, a reproduction function, a fast-forward 
function, a stop function, and an image recording 
function, respectively. Reference numeral 208 
designates an animation element . 

Reference numerals 209 to 212 designate icon 

25 elements. The icon elements 209 to 212 correspond to 
respective contents recorded on a storage medium to be 
reproduced by the DVC 100. If the remote control 130 
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selects one of these icon elements 209 to 212, the DVC 
100 starts a reproduction of a content corresponding to 
the icon element at 209 to 212 and the reproduced 
content is isochronously transferred to the DVC 100. 
5 Reference numeral 213 designates a focus. The 

focus 213 gives visual expressions to the elements 203 
to 212 by means of a highlighting display or a color 
change. A position of the focus 213 moves in response 
to an operation of the remote control 130. Processing 
10 of operating the focus in response to an operation of 
the remote control 130 is referred to as focal 
navigation. 

While the embodiment has been described by giving 
an example of displaying the operation panel of the DVC 
15 100 on the display unit 128 of the digital television 
120 in Fig. 2 , it is also possible to display an 
operation panel of the tuner 110. 

(Data structure of operation panel) 
Referring to Fig. 4 and Fig. 5, there are shown 
20 examples of data structures of operation panels 

according to this embodiment. Fig. 4 shows a diagram 
of a data structure of the operation panel 202. As 
shown in Fig. 4, GUI elements 202 to 212 forming the 
operation panel 202 have a tree structure routed from 
25 the panel element 202. Fig. 5 shows a diagram of a 
data structure of the panel element 202. 

In Fig. A, the panel element 202 retains a link 
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with text data showing a title and a functional 
description, a display position, a display size, and 
links with the elements 203 to 212. Each of the button 
elements 203 to 207 retains a link with text data 
showing a title and a functional description, a display 
position, a display size, and links with bit map image 
data corresponding to "depression" and "release" user 
actions . 

In addition, in Fig. 4, the animation element 208 
retains a link with text data showing a title and a 
functional description, a display position, a display 
size, and four links with bit map image data. Each of 
the icon elements 209 to 212 retains a link with text 
data showing a title and a functional description, a 
display position, a display size, and bit map image 
data corresponding to a "selection" user action. 

As shown in Fig. 4 and Fig. 5, an element ID is 
given to each GUI element and actual data linked 
therewith. 

While the data structures of the operation panels 
of the DVC 100 have been described in Fig. 4 and Fig. 
5, operation panels of the tuner 110 have the same data 
structures as shown in Fig. 4 and Fig. 5. 

(Description of remote control) 

Referring to Fig. 3, there is shown a diagram of 
an example of a remote control 130 according to this 
embodiment . 



In Fig. 3, there are shown operation keys 301 to 
305 for specifying movements of the focus 213. The 
reference numeral 301 designates a direction key for 
specifying a movement in an upper direction. The focus 
213 displayed on the operation panel moves upward 
according to an operation of the key 301. The 
reference numeral 302 designates a direction key for 
specifying a movement to the left. The focus 213 
displayed on the operation panel moves to the left 
according to an operation of the key 302. The 
reference numeral 303 designates a direction key for 
specifying a movement in a lower direction. The focus 
213 displayed on the operation panel moves downward 
according to an operation of the key 303. The 
reference numeral 304 designates a direction key for 
specifying a movement to the right. The focus 213 
displayed on the operation panel moves to the right 
according to an operation of the key 304. The 
reference numeral 305 designates a determination key 
for specifying a selection or a decision. 

Control signals corresponding to these keys 301 to 
305 are supplied to the control unit 122 via the remote 
control control unit 126 of the digital television 120. 
The control unit 122 controls movements of the focus 
213 on the basis of the control signals corresponding 
to the keys 301 to 304. 

The control unit 122 detects one of the 
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"depression", "release", and "selection" user actions 
on the basis of a control signal corresponding to the 
key 305. For example, if the focus 213 is adjusted to 
one of the button elements, the "depression" or 
5 "release" user action is detected. If the focus 213 is 
adjusted to one of the icon elements or the animation 
element, the "selection" user action is detected. The 
detected user action is converted to a USER_ACTION 

U5 command described later and then asynchronously 

03 

ii , 10 transferred to a controlled device. 



ill 



In the above, the "depression" means that the 
button element to which the focus 213 is adjusted has 
been put in a selected condition, the "release" means 
that the button element to which the focus 213 is 

nJ 

^i] 15 adjusted to has been put in a non- selected condition, 
H and " selection" means that the icon element or the 

animation element to which the focus 213 is adjusted 
has been put in a selected condition. 

A method of operating the operation panel by using 
20 the keys 301 to 305 for remotely operating the 

controlled device as set forth in the above is referred 
to as a focal navigation mode in this embodiment . 

Furthermore, in Fig, 3, there are shown operation 
keys 311 to 315 associated with operations 
25 corresponding to specific functions, where operations 
corresponding to the specific functions covered by the 
keys 311 to 315 depend upon an operation panel 
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currently under the operation. 

The reference numeral 311 designates an operation 
key for specifying a movement in a past, lower, or 
backward direction or a start or the like of rewinding. 
5 The reference numeral 312 designates an operation key 
for specifying a start or the like of reproduction. 
The reference numeral 313 designates an operation key 
for specifying a movement in a future, upper, or 
Q forward direction or a start or the like of a fast- 

03 10 forward operation. The reference numeral 314 

kfj designates an operation key for specifying a stop or 

HI the like of reproduction, rewinding, f ast-f ojrward or 

oi 

^" other operations. The reference numeral 315 designates 

"ri an operation key for specifying a start or the like of 

'It 15 recording. 

'^^ Control signals corresponding to these keys 311 to 

315 are supplied to the control unit 122 via the remote 
control control unit 126 of the DTV 110. The control 
unit 122 determines a currently selected operation 

20 panel and detects which key among the keys 311 to 315 
has been depressed on the basis of the control signal 
before generating a DIRECT_ACTION command described 
later. This DIRECT_ACTION command is asynchronously 
transferred to the controlled device. 

25 A method of remotely operating a specific function 

of the controlled device with a single-action button 
operation by using the keys 301 to 305 without any 
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operation of the operation panel is referred to as a 
direct action mode in this embodiment . The direct 
action mode requires less operation procedures in 
comparison with the focal navigation mode and enables 
5 the controlled device to be remotely operated even if 
the operation panel is not displayed. As set forth 
hereinabove, there are provided two operation methods, 
the focal navigation mode and the direct action mode in 
O this embodiment . 

gj 10 Furthermore, in Fig. 3, there are shown operations 

llj keys 321 to 324 for specifying operations on the DTV 

Si 

ijl 110. The reference numeral 321 designates an operation 

is 

J" key for specifying the power supply of the DTV 110 to 

TJ^i be turned on or off. The reference numeral 322 

15 designates an operation key for specifying an increase 
P of the volume of a sound or voice outputted by the DTV 

110. The reference numeral 323 designates an operation 
key for specifying a decrease of the volume of a sound 
or voice outputted by the DTV 110. The reference 
20 numeral 324 designates an operation key for calling a 
menu screen for adjusting various functions of the DTV 
110. On the menu screen, there are items for selecting 
a display or a non- display of an operation panel and 
items for selecting a desired operation panel. A user 
25 can change the display to the non-display or vice versa 
of the operation panel or can select an operation panel 
to be operated by operating these items. 
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Control signals corresponding to these keys 321 to 
324 are supplied to the control unit 122 via the remote 
control control unit 126 of the DTV 110. The control 
unit 122 controls the power supply to be turned on or 
off, a volxime adjustment, a menu screen call, or the 
like on the basis of the control signals. 

As set forth hereinabove, the remote control 130 
are provided with the first operation keys 301 to 305 
for operating the operation panel of the controlled 
device and the second operation keys 311 to 315 
directly associated with the specific functions of the 
controlled device, thereby enabling two types of 
operation methods to be provided. 

(Description of protocol) 

Referring to Fig. 6, there is shown a diagram of 
an example of a protocol of a remote control system 
according to this embodiment. By referring to Fig. 6, 
there is provided a description of a protocol between 
the digital television 120 which is a controller and 
the DVC 100 which is one of the controlled devices. 

The 1394 interface 121 of the digital television 
120 initially establishes an isochronous connection 
between the digital television 120 and the DVC 100 in 
accordance with a predetermined connection 
establishment procedure (601 in Fig. 6). The DVC 100 
isochronously transfers the stream data to the digital 
television 120 via this isochronous connection. 



Specifically, after accessing an IRM ( isochronous 
resource manager; the digital television 120 realizes 
this function here, for example) of a 1394 network and 
securing a channel and a band for the isochronous 
transfer, a peer-to-peer connection (a one-for-one 
logical connection) is established between an 
isochronous plug (which functions as a virtual output 
plug) of the 1394 interface 101 of the DVC 100 and an 
isochronous plug (which functions as a virtual input 
plug) of the 1394 interface 121 of the digital 
television 120. 

Next, the 1394 interface 121 of the digital 
television 120 establishes an asynchronous connection 
between the digital television 120 and the DVC 100 in 
accordance with a predetermined connection 
establishment procedure (602 in Fig. 6). The DVC 100 
asynchronously transfers GUI data of an operation panel 
to the digital television 120 via this asynchronous 
connection. 

The 1394 interface 121 of the digital television 
120 issues a GUI_UPDATE (start) command (603 in Fig. 
6). This GUI_UPDATE (start) command is used for 
specifying a start of transferring the operation panel 
of the DVC 100. 

The 1394 interface 121 of the digital television 
120 issues a PUSH_GUI_DATA command (604 in Fig. 6). 
This PUSH_GUI_DATA command is used for requesting the 



operation panel of the DVC 100. 

After receiving the PUSH_GUI_DATA command, the DVC 
100 reads out the GUI data of its own operation panel 
from the Panel Management Department 103 and supplies 
it to the 1394 interface 101. The 1394 interface 101 
sequentially transfers asynchronously this GUI data to 
the digital television 120 via the asynchronous 
connection (605 in Fig. 6). 

After receiving the GUI data of the DVC 100, the 
digital television 120 displays the operation panel of 
the DVC 100 on the basis of the GUI data and provides a 
user with a graphical operating environment (606 in 
Fig. 6). 

Specifically, the digital television 120 stores 
the GUI data of the DVC 100 in the Panel Management 
Department 124. The control unit 122 reads out this 
GUI data for each GUI element and supplies it to the 
drawing control unit 125. The drawing control unit 125 
analyzes a data structure of each GUI element , checks a 
position, a size, and contents for a display, and then 
controls the image composition unit 127 on the basis of 
the analysis result. The image composition unit 127 
synthesizes the GUI element supplied by the drawing 
control unit 125 with video data supplied by the stream 
data-processing unit 123 and digital-analog-converts 
the composed video data to a predetermined video 
signal. The display unit 128 provides a user with a 
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graphical operating environment as shown in Fig. 2, for 
example, by displaying this video signal. 

After displaying the operation panel of the DVC 
100, the digital television 120 enables the above two 
5 operation methods (namely, the focal navigation mode 
and the direct action mode) to be executed. 

In the focal navigation mode, a focal navigation 
is performed in accordance with a user operation and a 
user action ("depression", "selection", etc.) permitted 
Sf 10 for each GUI element is accepted. On the other hand, 
^ in the direct action mode, a direct action is accepted 

in accordance with any of the keys 311 to 315 of the 
remote control 130 independently of whether there is a 
C3 display of the operation panel. 

yj 

HJ 15 If the user action is detected, the 1394 interface 

□ 121 issues a USER_ACTION command (607 in Fig. 6). This 

USER_ACTION command is used for indicating what user 
action has occurred for a certain GUI element . There 
will be described hereinafter a case of issuing the 
20 USER_ACTION command indicating that a "depression" user 
action has occurred to the icon element 209, for 
example . 

After receiving the USER_ACTION command, the DVC 
100 controls the DVC unit 104 on the basis of a GUI 
25 element and a user action specified by the USER_ACTION 
command (608 in Fig. 6). For example, the DVC unit 104 
retrieves and reproduces contents corresponding to the 



icon element 209 from the storage medium 105 and 
supplies these contents as stream data to the 1394 
interface 101. 

If the GUI of the operation panel changes in 
accordance with the USER_ACTION command, the DVC 100 
reads out the GUI data corresponding to the change 
portion from the Panel Management Department 103 and 
transfers the GUI data to the digital television 120 
via the asynchronous connection (609 in Fig. 6) . 

The DVC 100 transfers the contents corresponding 
to the icon element 209 via the isochronous connection 
(610 in Fig. 6 ) . 

After receiving the GUI data and the contents 
corresponding to the icon element 209, the digital 
television 120 updates the operation panel of the DVC 
100 on the basis of the received GUI data and 
synthesizes the updated operation panel with the 
received contents before displaying (611 in Fig. 6). 

On the other hand, if the direct action is 
detected, the 1394 interface 121 issues a DIRECT_ACTION 
command (612 in Fig. 6). 

Receiving the DIRECT_ACTION command, the DVC 100 
controls specific functions specified by the 
DIRECT_ACTION command (613 in Fig. 6). 

In the above, if the specific function specified 
by the DIRECT_ACTION command is invalid for the DVC 100 
which is a controlled device, in other words, if it is 
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not included in services provided by the DVC 100 or 
cannot be temporarily executed due to an internal 
condition, an error response indicating a non- 
executable condition is returned. 
5 If the GUI of the operation panel changes in 

accordance with the DIRECT_ACTION command, the DVC 100 
reads out the GUI data corresponding to the change 
portion from the Panel Management Department 103 and 
□ transfers the GUI data to the digital television 120 

Qj 10 via the asynchronous connection (614 in Fig. 6). 

The digital television 120 updates the operation 

\j 

panel of the DVC 100 on the basis of the received GUI 
data (615 in Fig. 6). In the above, if the operation 

H panel is in a non-display condition, the digital 

15 television 120 displays the updated operation panel on 

O the display screen until the user specifies a non- 

display condition or until a predetermined period of 
time elapses. This configuration enables the result of 
the operation to be graphically expressed on the 
20 operation panel even if an operation is executed in the 
direct action mode. In addition, a user can visually 
confirm which function of the DVC 100 has been actually 
operated, by which the current operating condition can 
always be grasped. 
25 The digital television 120 and the DVC 100 

repeatedly execute the procedure in 604 to 615 to 
accept the operations in the focal navigation mode and 
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the direct action mode. This enables the user to use 
two operation methods together, by which a plurality of 
operations can be continuously performed in a short 
time. Therefore, for example, it is effective to 
5 repeat the reproduction, fast-forward, reproduction, 
and fast-forward operations in a short time. In 
addition, for example, it is also effective to 
continuously operate GUI elements located apart from 

O each other on the operation panel in a short time. 

pj 10 If the operation in the focal navigation mode and 

the direct action mode is terminated, the 1394 

I|S interface 121 of the digital television 120 issues a 

m 

y GUI_UPDATE (stop) command (616 in Fig. 6). 

J"f, Afterward, the asynchronous connection is 

15 disconnected and then the isochronous connection is 
O disconnected (617 and 618 in Fig. 6). 

While there has been described the protocol 
between the digital television 120 and the DVC 100 in 
Fig. 6, the present invention is not limited to this . 
20 The tuner 110 that is another controlled device can 

communicate with the digital television 120 in the same 
manner as for this protocol. Furthermore, a 
communication between the digital television 120 and 
the DVC 100 can be executed in parallel with a 
25 communication between the digital television 120 and 
the tuner 110. 

Next, the above GUI_UPDATE command, the 
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USER_ACTION command, and the DIRECT_ACTION command will 
be described below. The control commands are 
transferred on the basis of the function control 
protocol stipulated in the IEC61883-1. 
5 Fig. 7A shows a control command stipulated by the 

function control protocol and Fig. 7B shows a frame 
structure of a response corresponding thereto. " ctype" 
and "response" indicate a command type and a response 
C3 type, respectively- The " subunit__type" and 

m 10 " subunit_ID" are used for specifying a sub-unit to be 

[i^ controlled. "opcode" indicates controlled contents and 

/jif operand[l] to operand[n] are defined according to the 

contents of opcode. 

(Description of GUI_UPDATE command) 

xs- J 
: 

15 Referring to Fig. 8A, there is shown a diagram of 

O assistance in explaining a command frame of the 

GUI_UPDATE command according to this embodiment, 
"source plug" is stored in operand[l] and " subf unction" 
indicating an instruction for the controlled device is 
20 stored in operand[3]. 

The content of " subfunction" is described below by 
using Fig. SB. An "open" code is used for giving an 
instruction on a start of a remote operation with an 
operation panel (hereinafter, simply referred to as a 
25 remote operation), A "close" code is used for giving 
an instruction on an end of the remote operation. 

A " start" code is used for giving an instruction 



on a start of transferring GUI data forming the 
operation panel. A "stop" code is used for giving an 
instruction on a stop of transferring the GUI data. 
The controlled device transmits GUI data depending upon 
changes in a device condition or in response to a 
result of an operation until this GUI_UPDATE (stop) 
command is received . 

A "restore" code is used for giving an instruction 
on a return of an asynchronous connection after a bus 
reset occurs in the network 1 . A " change" code is used 
for giving an instruction on a change of a display 
range of an operation panel. 

(Description of PUSH_GUI_DATA command) 

Referring to Fig, 9A, there is shown a diagram of 
assistance in explaining a command frame of the 
PUSH_GUI_DATA command according to this embodiment. A 
plug number obtained by the GUI_UPDATE (open) command 
is stored in operand[0]. An "indicator" is stored in 
operand[5]. Element IDs for identifying GUI elements 
are stored in operand[6] to operand[9]. 

The contents of the "indicator" will be described 
below by using Fig. 9B. The "indicator" comprises 
" level" and " with_data" . A code is set to " level" for 
specifying a range of GUI data requested of the 
controlled device by the controller. If actual data (a 
text, an image, a sound, etc.) of the GUI element 
specified by using the element ID and " level" , " 1" is 
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stored in "with_data" . 

The contents of " level" will be described by using 
Fig. 9C. An "itself" code is used for requesting the 
GUI element itself specified by the element ID, An 
5 " itself and next level" code is used for requesting the 
GUI element specified by the element ID and a GUI 
element belonging to the next level of the GUI element 
and an " all level" code is used for requesting all the 
GUI elements. 
10 (Description of USER_ACTION command) 

Referring to Fig. 10, there is shown a diagraun of 
assistance in explaining a command frame of a 
-55 USER_ACTION command according to this embodiment. In 

O operand [ 0 ] , operand [ 3 ] to operand [ 6 ] , operand [ 7 ] , and 

15 operand[8] or after, " source plug" , " element_id" , 

" action_type" , and "data" are stored, respectively. 

The " element_id" code is used for indicating an 
element ID for identifying an GUI element. The 
" action_type" code is used for specifying a user action 
20 such as a depression, a release, and a selection. 
(Description of DIRECT_ACTION command) 
Referring to Fig. 11, there is shown a diagram of 
assistance in explaining a command frame of a 
DIRECT__ACTION command according to this embodiment. In 
25 operand[0], operand[2], and operand[31 or after, 

" action_code" , " action_type" , " action_data„length" , and 
" action_specif ied_data" are stored, respectively. 



nj 
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The " action__code" code is used for indicating a 
specific function* The " action_type" code is used for 
indicating how a specific function is operated* The 
" action_data_length" code is used for indicating a data 
5 length of " action_specif ied_data" . The 

" action_specif ied„data" code is used for indicating 
data inherent in the " action_code" code . 

Referring to Fig. 12, there is shown a diagram of 
5=:^ an example of the " action_code" code shown in Fig. 11. 

10 If the currently selected operation panel is an 

Li I 

V operation panel of the DVC 100 in this embodiment, the 

^ " action_code" corresponding to the key 311 is " 28h" , 

03 which indicates rewinding of the storage medium 105, 

O for example. The " action_code" corresponding to the 

ffJ 15 key 312 is " 24h" , which indicates a start of 
reproduction of the storage medium 105. The 
" action__code" corresponding to the key 313 is " 29h , 
which indicates fast -forwarding of the storage medium 
105. The " action_code" corresponding to the key 314 is 
20 " 25h" , which indicates a stop of an operation of the 

storage medium 105. The " action_code" corresponding to 
the key 315 is " 27h" , which indicates a start of 
recording into the storage medium 105. 

On the other hand, if the currently selected 
25 operation panel is an operation panel of the tuner 110, 
the " action_code" corresponding to the key 311 is 
" 2Bh" , which indicates a movement in a lower direction 
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of a receiving channel. The " action_code" 
corresponding to the key 312 is " 06h" , which indicates 
a display of EPG (electric program guide). The 
" action_code" corresponding to the key 313 is " 2Ch" , 
5 which indicates a movement in an upper direction of the 
receiving channel . 

(Description of processing procedure of controlled 
device ) 

Referring to Fig. 13, there is shown a diagram of 

10 assistance in explaining a processing procedure of a 

controlled device according to this embodiment . While 
a processing procedure of the DVC 100 that is one of 
the controlled devices is described here, the tuner 110 
can also be processed as described below. 

15 In step SlOl the 1394 interface 101 receives a 

control command corresponding to a user remote control 
operation. This type of control commands include the 
above-described USER„ACTION command, the above- 
described DIRECT_ACTION command, and other control 

20 commands. 

Next, in step S102, the control unit 102 
determined whether the received control command is the 
USER_ACTION command. If it is the USER_ACTION command, 
step S103 and subsequent processing is executed; 

25 otherwise, step S109 and subsequent processing is 
executed. 

In step S103, the control unit 102 checks contents 



of the received USER_ACTION coiranand and determines 
whether it is an executable action. If it is 
executable, the control unit 102 executes processing of 
step S104; otherwise, the control unit 102 executes 
processing of step S108. In the step S108, an error 
response is generated for indicating that the received 
USER_ACTION command cannot be processed and the error 
response is returned to the controller. 

In step S104, the 1394 interface 101 normally 
receives the USER__ACTION command and returns a response 
indicating that the command is executable to the 
controller. 

Next, in step S105, the control unit 102 controls 
a function corresponding to an operation panel 
specified by the " element_id" code of the USER_ACTION 
command according to the " action_type" code of the 
USER_ACTION command. 

In step S106, the control unit 102 determines 
whether the operation panel operated by the controller 
should be changed together with an execution of the 
USER__ACTION command. If it should be changed, 
processing of step S107 is executed; otherwise, the 
processing control returns to the step SlOl to receive 
the next control command. 

In the step S107, the 1394 interface 101 transmits 
the GUI data of a portion to change to the controller. 
Afterward, the controlled device returns to the step 



SlOl to receive the next control command. 

On the other hand, the control unit 102 determines 
whether the received control command is a DIRECT__ACTION 
command in step S109. If it is the DIRECT_ACTION 
command, step SllO and subsequent processing is 
executed; otherwise, step S116 and subsequent 
processing is executed- In step S116, processing 
corresponding to the received control command is 
executed and then the processing control returns to the 
step SlOl to receive the next control command. 

In the step SllO, the control unit 102 checks 
contents of the received DIRECT_ACTION command and 
determines whether it is an executable action. If it 
is executable, the control unit 102 executes processing 
of step Sill; otherwise, the control unit 102 executes 
processing of step S115. In the step S115, an error 
response is generated for indicating that the received 
DIRECT_ACTION command cannot be processed and the error 
response is returned to the controller. 

In step S112, the 1394 interface 101 normally 
receives the DIRECT_ACTION command and returns a 
response indicating that the command is executable to 
the controller. 

Next, in the step 8112, the control unit 102 
controls a function directly specified by the 
" action_code" code of the DIRECT_ACTION command 
according to the " action_type" code of the 



- 37 - 



DIRECT_ACTION command. 

In Step S113, the control unit 102 determines 
whether the operation panel displayed on the controller 
should be changed together with an execution of the 
5 DIRECT_ACTION command. If it should be changed, 

processing of step S114 is executed; otherwise, the 
processing control returns to the step SlOl to receive 
the next control command . 

p In the step S114, the 1394 interface 101 transmits 

'41 

03 10 the GUI data of a portion to change to the controller. 

Ifi Afterward, the controlled device returns to the step 

ss? SlOl to receive the next control command. 

In the above processing procedure, each controlled 

T^i device can execute both of the USER_ACTION command and 

yj 

15 the DIRECT_ACTION command, and if an operation panel 

^3 ■ 

G changes according to these control commands, GUI data 

of a portion corresponding to the change can be 
transmitted to the controller. 

(Description of processing procedure of 
20 controller) 

Referring to Fig. 14, there is shown a diagram of 
assistance in explaining a processing procedure of the 
digital television 120 that is a controller according 
to this embodiment . 
25 In step S201, the control unit 122 selects an 

operation panel to be operated according to a user 
remote control operation and displays the operation 
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panel selected in accordance with the above -described 
protocol on the display unit 128. 

In step S202, the remote control control unit 126 
receives a control signal from the remote control 130. 
5 In step S203, the control unit 122 checks the 

received control signal and determines whether it is an 
operation of a focus . If it is an operation of the 
focus, the above -described focal navigation is executed 

O in step S208. Otherwise, processing of step S204 is 

pi 10 executed. 

J5 In step S204, the control unit 122 checks the 

received control signal and determines whether it is an 
^' operation in the user action mode. If it is an 

operation in the focal navigation mode, a USER__ACTION 
^jf 15 command corresponding to the received control signal is 

Q generated and processing of step S205 is executed. 

Otherwise, processing of step S209 is executed. 

In step S205, the 1394 interface 121 issues the 

USER_ACTION command to a controlled device 
20 corresponding to the currently operated operation 

panel . 

In step S206, the 1394 interface 121 determines 
whether the GUI data of a portion to change on the 
operation panel has been received. If the GUI data of 
25 a portion to change is received, the operation panel is 
changed on the basis of the GUI data in the step S206. 
If the operation panel is in the non-display condition 
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at this time, the changed operation panel is displayed 
on the display screen until a user specifies the non- 
display condition or until a predetermined period of 
time elapses. 

5 In step S209, the control unit 122 checks the 

received control signal and determines whether it is an 
operation in the direct action mode. If it is the 
operation in the direct action mode, the DIRECT_ACTION 
O command corresponding to the received control signal is 

m 10 generated and processing of step S210 is executed. 

If^ Otherwise, a control of the digital television 120 or 

the like is executed in step S213. 
~" In the step S210, the 1394 interface 121 issues a 

r;1 DIRECT__ACTION command to a controlled device 

K : ii 

"jf 15 corresponding to the currently operated operation 

panel . 

In step S211, the 1394 interface 121 determines 
whether GUI data of a portion to change on the 
operation panel has been received. If the GUI data of 
20 a portion to change is received, the operation panel is 
changed on the basis of the GUI data in step S212- If 
the operation panel is in the non-display condition at 
this time, the changed operation panel is displayed on 
the display screen until the user specifies the non- 
25 display condition or until a predetermined period of 
time elapses . 

In the above-described processing procedure, the 



controller can remotely operate a controlled device 
using two operation methods together. In addition, if 
either of the operation methods is used for an 
operation, a result of the operation can be graphically 
expressed on the operation panel. Furthermore, the 
user can visually confirm which function of the 
controlled device has been actually operated, by which 
the current operating condition can always be grasped. 

As set forth hereinabove, according to this 
embodiment, two operation methods of the focal 
navigation mode and the direct action mode are used 
together, by which it becomes possible to flexibly cope 
with operating environments for a plurality of types of 
electronic devices and to remotely operate these 
electronic devices in a simple operation procedure. 

Furthermore, according to this embodiment, two 
operation methods of the focal navigation mode and the 
direct action mode are used together, by which a user 
can execute both of the first operation of selecting a 
desired function while viewing an operation panel and 
the second operation of directly selecting a desired 
function without operating an operation panel. This 
reduces a complication of an operation, thereby 
significantly improving facility in operation. 

In addition, according to this embodiment, a 
plurality of operations can be continuously performed 
in a short time by using these two operation methods 
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together. 

Furthermore, according to this embodiment, results 
of operations in the two operation methods are always 
reflected on an operation panel, by which a user easily 
5 check what condition the operation is in at present so 
as to prevent the user from being confused. 

Still further, according to this embodiment, a 
function corresponding to an operation key for 
O realizing the direct action mode, can be changed for 

nj 10 each operation panel. This makes it possible to cope 

with various types of controlled devices flexibly 
without increasing the number of operation keys . 



U1 



The invention may be embodied in other specific 
forms without departing from essential characteristics 

15 thereof - 

For example, while the digital television 120 has 
been described as an example of a controller in this 
embodiment, the controller is not limited to the 
digital television. Only if it is an electronic device 

20 having an operation apparatus equivalent to the remote 
control 130 and a display, any device can be used for 
it. 

In addition, while the DVC 100 and the tuner 110 
have been described as examples of controlled devices 
25 in this embodiment, controlled devices are not limited 
to these devices. They can be either of audio-visual 
devices other than the DVC 100 and the tuner 110 (for 
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example, a digital video recorder, a set -top box, etc) 
and household electric appliances . 

Therefore, the above-mentioned embodiments are 
merely examples in all respects , and must not be 
construed to limit the invention. 

The scope of the present invention is defined by 
the scope of the appended claims, and is not limited at 
all by the specific descriptions of this specification. 
Furthermore, all the modifications and changes 
belonging to equivalents of the claims are considered 
to fall within the scope of the present invention. 



